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ABSTRACT 

Faculty in a large, urban school of engineering designed a longitudinal study to assess the critical 
thinking skills of undergraduate students as they progressed through the engineering program. 
The Paid-Elder critical thinking framework was used to design course assignments and develop a 
holistic assessment rubric. The curriculum was re-designed to include deliberate teaching of 
critical thinking and assessment in at least one key course for every student each year of their 
undergraduate curriculum. The critical thinking scores for seniors using the holistic rubric were 
significantly higher than their baseline critical thinking scores as freshmen (p = .004). This case- 
study can serve as an exemplar for other units, departments, or programs to model or replicate. 
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INTRODUCTION 


yy ' n 2007, the university began implementing a multi-year plan to enhance undergraduate students' critical 
thinking skills in all undergraduate programs as part of the Southern Association of Colleges and Schools 
(SACS) regional reaccreditation process. To focus the reaccreditation plan, the university selected 
Scriven and Paul’s (1987) definition that critical thinking is the intellectually disciplined process of actively and 
skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or 
generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action. 
Faculty, administrators and staff responsible for leading the initiative reviewed several critical thinking models but 
unanimously agreed to select the Paul-Elder critical thinking framework (Paul and Elder, 2009) to provide 
intentional and consistent language across undergraduate programs because of the Paul-Elder framework’s 
comprehensiveness, discipline neutral terminology and extensive high quality resources. The Engineering School 
embraced the process because engineering faculty felt, and research supports that critical thinking is foundational to 
engineering education and to engineering practice (Alfrey and Cooney, 2009; Rogers, 2006). Although critical 
thinking and assessment of critical thinking using rubrics have been discussed in the engineering education literature 
(Berge and Flora, 2010; Goulter et al., 2009; Ralston and Bays, 2010), the studies have primarily related to a single 
course, not complete curriculum redesign and assessment. Although program assessment of critical thinking has 
been undertaken in other fields (Bensley and Murtagh, 2012; Cavaliere and Mayer, 2012; McKitrick and Barnes, 
2012), the only published discussion of program assessment of critical thinking for engineering was the recent work 
of Eppes et al (2012). Their work presents an assessment framework for five liberal education outcomes, including 
critical thinking, in the University of Hartford’s engineering and technology programs, but gives results from only a 
pilot study involving one capstone design course. 


This paper presents the first cohort results from a comprehensive curricular redesign and assessment of 
critical thinking at a large, urban engineering college with 14 ABET (Accreditation Board for Engineering and 
Technology) accredited programs. In fall 2008, faculty at the Engineering School began a longitudinal, three cohort 
educational research study to incorporate the Paul-Elder critical thinking framework intentionally and transparently, 
across the undergraduate engineering curriculum. Faculty developed and implemented a plan for assessing program 
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results that operates as an ongoing feedback loop consisting of implementation, then assessment, followed by 
revision, then re-assessment for evaluation of critical thinking skills, much as the ongoing Accreditation Board for 
Engineering and Technology (ABET) assessments function for engineering program curricula. First, the Paul-Elder 
Framework is briefly reviewed and a short summary of the relationship of critical thinking to ABET outcomes is 
discussed. Then, the baseline data analysis for the first cohort of students that completed the entire redesigned 
curriculum that included critical thinking each year of the undergraduate program is explained. Details of the 
longitudinal study development analysis of the freshman course artifacts (baseline and course critical thinking 
assignments) and associated faculty scoring sessions for all three cohorts can be found in (2011, Inquiry) which 
could provide a model for other programs. 

PAUL-ELDER CRITICAL THINKING LRAMEWORK 

The Paul-Elder (2009) critical thinking framework has a formal structure and is a discipline-neutral 
schema. Figure 1 depicts the Paul-Elder critical thinking framework. The framework depicts critical thinking by 
applying Universal Intellectual Standards to the evaluation of typical Elements of Thought, with the goal of 
developing certain Essential Intellectual Traits in the thinker. 



Figure 1: The Paul-Elder Critical Thinking Framework, Adapted from Paul and Elder (2009) 

The eight Elements of Thought are the parts or fundamental structures of thought, which are the essential 
dimensions when all reasoning occurs in all persons at any time. The Elements of Thought work together in a 
nonlinear interrelationship to shape reasoning and provide a general logic to the use of thought. There is an intimate 
overlap among all of the elements by virtue of their interrelationship. Fundamental to critical thinking is the ability 
to assess the quality of the reasoning. Assessing critical thinking involves consistently taking apart and examining 
thinking with respect to the Universal Intellectual Standards of quality. Unlike the complete list of eight Elements of 
Thought, the ten Universal Intellectual Standards presented in Figure 1 are considered minimal or fundamental 
intellectual standards within a wider variety from which to choose. The Universal Intellectual Standards are criteria 
for assessing the quality of reasoning that serve as guides to better reasoning. The Essential Intellectual Traits are 
developed by consistently applying the Universal Intellectual Standards to the Elements of Thought. The Essential 
Intellectual Traits are tendencies or commitments towards the trait and not skills or abilities, (Paul and Elder, 2009). 
The framework allows for the analysis and evaluation of thought, but more importantly, it provides a common 
vocabulary for those who want to discuss, evaluate, or teach critical thinking. 
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Engineering programs must demonstrate that their students attain the following outcomes: (a) an ability to 
apply knowledge of mathematics, science, and engineering, (b) an ability to design and conduct experiments, as well 
as to analyze and interpret data, (c) an ability to design a system, component, or process to meet desired needs 
within realistic constraints such as economic, environmental, social, political, ethical, health and safety, 
manufacturability, and sustainability, (d) an ability to function on multidisciplinary teams, (e) an ability to identify, 
formulate, and solve engineering problems, (f) an understanding of professional and ethical responsibility, (g) an 
ability to communicate effectively, (h) the broad education necessary to understand the impact of engineering 
solutions in a global, economic, environmental, and societal context, (i) a recognition of the need for, and an ability 
to engage in life-long learning, (j) a knowledge of contemporary issues, (k) an ability to use the techniques, skills, 
and modern engineering tools necessary for engineering practice. Although the term critical thinking is not used in 
any ABET outcome, they all require “the intellectually disciplined process of actively and skillfully conceptualizing, 
applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, 
experience, reflection, reasoning, or communication, as a guide to belief or action”, i.e. critical thinking 
( http://www.abet.org/DisplayTemplates/DocsHandbook.aspx?id=3143 ). 

METHODS 

The purpose of this study was to assess the development of critical thinking skills in undergraduate 
engineering students. The research question was, “How do the critical thinking skills of undergraduate engineering 
students change as they progress through the engineering program?” The study was a descriptive, longitudinal study 
of three engineering student cohorts as they progressed through the four year undergraduate program. The study was 
approved by the university’s Institutional Review Board. 

PARTICIPANTS 

Undergraduate engineering students were eligible to participate in the study if they were at least 18 years of 
age as freshmen and would complete the undergraduate engineering program in four years. A convenience sample of 
187 undergraduate engineering freshmen students consented to participate in the study (51% consent rate). For the 
sample, the students mean high school grade point average was 3.71 (SD 0.31) and ACT composite score was 28.11 
(SD 2.9). The majority of the sample was Caucasian (83%) males (79%). 

Forty-nine of the sample (26% of those who consented) completed the undergraduate engineering program 
in four years and are the unit of analysis for the longitudinal study. The greatest decrease in the study sample 
occurred from the freshman to sophomore year where 65 of the freshmen subjects continued to the sophomore year 
(35% of original consent). This initial decrease was primarily due to first year attrition since students often transfer 
to other majors their first year. The continued decrease in sample size was primarily due to students changing majors 
within engineering failing to stay in the regular sequence of program courses due to dropping courses, not taking a 
full load, or other reasons. 

DATA COLLECTION 

The process started with a core group of committed faculty who recognized that although most all 
engineering courses required students to think critically, faculty were not specifically teaching students how to think 
critically. Faculty typically “ demonstrated !” critical thinking as they solved problems or discussed ethical or 
technical issues, rather than explicitly using a common language and framework to thoroughly discuss the “analyze 
and evaluate ” process of critical thinking so clearly articulated in (Nosich, 2012). Engineering faculty developed or 
revised a critical thinking assignment for a selected course in each year of the undergraduate program. The 
assignments were authentic assessments embedded within the courses as part of the grading requirements and direct 
assessments of the students’ critical thinking abilities for course specific content. Student responses to the 
assignments (artifacts) were copied before grading, all identifying information was removed, and a unique study 
number was placed on the artifacts for tracking across the four-year study time period. 
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Several faculty members in every department embraced the project to the extent that we are convinced 
every student was thoroughly exposed to the Paul-Elder critical thinking framework in at least one course per year. 
The specific critical thinking assignment for a course each year that all engineering students took was: 

• Freshman: For “Introduction to Engineering” course, a case-study was used to introduce various aspects of 
engineering before the Paul-Elder critical thinking framework was intentionally discussed in the course. 
The case study served as the baseline or “pre” assessment. 

• Sophomore: The “Differential Equations” course’s artifact was a one page assignment that required 
students to answer the question “If you had time to teach only one method to solve differential equations, 
which method would you pick and why?” Faculty reviewed the Paul-Elder critical thinking framework 
before this assignment was submitted. 

• Junior: The artifact collected from third year students was their cooperative internship report summary 
which asked students to critically reflect on their primary responsibilities. The cooperative internship 
seminar classes spent time formally reviewing the Paul-Elder critical thinking framework, as well as 
modeling for students how critical thinking skills could be used in interviewing and in engineering practice. 

• Senior: Artifacts from fourth year courses from the seven degree-granting departments were written 
summaries from either capstone design projects or other individual assignments requiring critical analysis. 
In the fourth year, some faculty teaching courses where artifacts were collected made overt efforts to teach 
the Paul-Elder critical thinking framework, and others simply provided review materials. 

ANALYSIS 

No existing critical thinking rubric was found that incorporated the Paul-Elder critical thinking framework 
for engineering. The engineering faculty decided to construct a four-point, holistic critical thinking rubric based on 
the Paul-Elder critical thinking framework for use with the research project (Ralston and Bays, 2010). Figure 2 
presents the engineering holistic critical thinking rubric. 


University of X 
X School of Engineering 
Holistic Critical Thinking Rubric* 

Consistently does all or most of the following: 

4 Clearly identifies the purpose including all complexities of relevant questions. 

Accurate, complete information that is supported by relevant evidence. 

Complete, fair presentation of all relevant assumptions and points of view. 

Clearly articulates significant, logical implications and consequences based on relevant evidence. 

3 Clearly identifies the purpose including some complexities of relevant questions. 

Accurate, mostly complete information that is supported by evidence. 

Complete, fair presentation of some relevant assumptions and points of view. 

Clearly articulates some implications and consequences based on evidence. 

2 Identifies the purpose including irrelevant and/or insufficient questions. 

Accurate but incomplete information that is not supported by evidence. 

Simplistic presentation that ignores relevant assumptions and points of view. 

Articulates insignificant or illogical implications and consequences that are not supported by evidence. 

1 Unclear purpose that does not includes questions. 

Inaccurate, incomplete information that is not supported by evidence. 

Incomplete presentation that ignores relevant assumptions and points of view. 

Fails to recognize or generates invalid implications and consequences based on irrelevant evidence. 

*Based on the Paul-Elder critical thinking framework _ 

Figure 2: Engineering Holistic Critical Thinking Rubric 

A total of 15 engineering faculty volunteered to score the student artifacts throughout the research project. 
Faculty received a stipend to spend on any academic expense as a compensation for the time the scoring session 
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would take over the multi-year length of the research project. Engineering faculty were trained to score the blinded 
artifacts using the holistic critical thinking rubric. At least two faculty independently scored each artifact. Each 
faculty rater was given a file with the blinded student artifacts, assignment description, critical thinking rubric, and 
form to record their scores. If there was greater than a one point discrepancy between the two faculty rater scores, a 
third faculty scored the artifact. Final critical thinking scores for each artifact were determined by averaging the two 
or three blinded, independent faculty rater scores. The intraclass correlation coefficient (ICC) was used to assess the 
consistency of the 42 paired faculty rater scores across the four assessment periods. The ICCs ranged from 0.94 to 
-0.811. Forty five percent of the ICCs were greater than or equal to 0.4 (n = 19) and 6 (32%) of those were 
significant at the 0.05 level. The 8 negative ICCs (19%) reflected a high within-subjects variance for artifacts rated 
by those faculty pairs (Bartko, 1976; Shrout and Fleiss, 1979). The number of negative ICCs occurred in different 
faculty pairs. The number of artifacts that necessitated a third rater averaged 17% across the four assessment 
periods. 

FINDINGS 

A repeated measures ANOVA was used to assess the change in subject’s critical thinking scores over time. 
The assumption of sphericity, relatively equal variances, was evidenced by Mauchly’s Test of Sphericity (W = .893, 
Y 2 (5) = 4.715, p = 0.452) so no corrections were applied to the F-ratio computations. The mean critical thinking 
rubric score for the freshman artifact was 2.599 (SD 0.107), sophomore 2.625 (SD 0.090), junior 2.833 (SD 0.088), 
and senior 3.152 (SD 0.098). Figure 3 depicits the critical thinking scores for the four assessment points. 



There was a statistically significant increase in critical thinking scores over the four years (F(3) = 6.693, p 
= .000). Specifically, the critical thinking score for the senior artifact was significantly higher than the freshman (p = 
.004) and the sophomore (p = .002) critical thinking scores. Table 1 reports the Bonferroni comparison for the 
critical thinking scores. 


Table 1: Bonferroni Comparison of Critical Thinking Scores 


Comparison 

Mean Score Difference 

Std. Error 

Sig. 

95%CI 

Lower Upper 

Bound Bound 

Freshman with Sophomore 

.026 

.147 

1.00 

.433 

.380 

Freshman with Junior 

.235 

.147 

.707 

.641 

.172 

Freshman with Senior 

.553 

.152 

.004 

.973 

.133 

Sophomore with Junior 

.208 

.113 

.435 

.521 

.105 

Sophomore with Senior 

.527 

.135 

.002 

.901 

.152 

Junior with Senior 

.318 

.140 

.169 

.706 

.069 
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Faculty involvement in the creation of critical thinking assignments across the curriculum and assessment 
of student responses to the critical thinking assignment can strengthen the students’ development of critical thinking 
abilities. The significant increase in the seniors’ critical thinking scores from the holistic rubric is consistent with 
seniors from the same year whose score on the Collegiate Assessment of Academic Proficiency (CAAP) increased 
2.4 points from baseline to 68.3, which is especially notable since the national average declined 1.4 points to 60.6 
The increase in undergraduate engineering students’ critical thinking abilities is an encouraging finding for the 
engineering profession and public who need engineers who can think critically. An area for refinement is the 
interrater reliability that can be enhanced by intentional pairing of raters, providing a review of the rating 
expectations, limiting the rating session to two hours to reduce fatigue, and ensuring each rater pair scores at least 10 
artifacts. The primary limitation of the study was the small, convenience sample. Further improvement and 
refinement will be achieved as engineering faculty strive to increase the number of courses that intentionally focus 
on critical thinking. 

This model for developing assignments and assessing student responses can be duplicated in other 
disciplines interested in program assessment of critical thinking. Results from the two remaining cohorts will guide 
further refinement of this engineering school’s assessment of students’ critical thinking skills. However, these 
results support that the overt teaching of critical thinking using the Paul-Elder framework has a positive impact on 
engineering students. 
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